Jul 24
5
Do you use a CDN for some or all of your website and you want to manage just one robots.txt file, instead of both the CDN’s robots.txt file and your main site’s robots.txt file? Gary Illyes from Google said that all you need to do is redirect your main robots.txt to the CDN and control it on the CDN.
Follow this link:
Google: Using A CDN & Want One Robots.txt File, Redirect Yours To The CDN